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Abstract. We investigate logics and equivalence relations that capture the qualitative 
behavior of Markov Decision Processes (MDPs). We present Qualitative Randomized Ctl 
(Qrctl): formulas of this logic can express the fact that certain temporal properties hold 
over all paths, or with probability or 1, but they do not distinguish among intermediate 
probability values. We present a symbolic, polynomial time model-checking algorithm for 
Qrctl on MDPs. 

The logic Qrctl induces an equivalence relation over states of an MDP that we call 
qualitative equivalence: informally, two states are qualitatively equivalent if the sets of 
formulas that hold with probability or 1 at the two states are the same. We show that 

for finite alternating MDPs, where noiidcterministic and probabilistic choices occur in dif- 
ferent states, qualitative equivalence coincides with alternating bisimulation, and can thus 
be computed via efficient partition-refinement algorithms. On the other hand, in non- 
alternating MDPs the equivalence relations cannot be computed via partition-refinement 
algorithms, but rather, they require non-local computation. Finally, we consider Qrctl*, 
that extends Qrctl with nested temporal operators in the same manner in which Ctl* ex- 
tends Ctl. We show that Qrctl and Qrctl* induce the same qualitative equivalence on 
alternating MDPs, while on non-alternating MDPs, the equivalence arising from Qrctl* 
can be strictly finer. We also provide a full characterization of the relation between qual- 
itative equivalence, bisimulation, and alternating bisimulation, according to whether the 
MDPs are finite, and to whether their transition relations are finitely-branching. 
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1. Introduction 

Markov decision processes (MDPs) provide a model for systems exhibiting both probabihs- 
tic and nondeterministic behavior. MDPs were originally introduced to model and solve 
control problems for stochastic systems: there, nondeterminism represented the freedom in 
the choice of control action, while the probabilistic component of the behavior described 
the system's response to the control action |Ber95] . MDPs were later adopted as models 
for concurrent probabilistic systems, probabilistic systems operating in open environments 
Seg95 , and under-specified probabilistic systems |BdA95l ldA97aj . 



Given an MDP and a property of interest, we can ask two kinds of verification questions: 
quantitative and qualitative questions. Quantitative questions relate to the numerical value 
of the probability with which the property holds in the system; qualitative questions ask 
whether the property holds with probability or 1. Examples of quantitative questions 
include the computation of the maximal and minimal probabilities with which the MDP 
satisfies a safety, reachability, or in general, w-regular property [BdA95j : the corresponding 
qualitative questions asks whether said properties hold with probability or 1. 

While much recent work on probabilistic verification has focused on answering quanti- 
tative questions, the interest in qualitative verification questions predates the one in quan- 
titative ones. Answering qualitative questions about MDPs is useful in a wide range of 
applications. In the analysis of randomized algorithms, it is natural to require that the 
correct behavior arises with probability 1, and not just with probability at least p for some 
p < 1. For instance, when analyzing a randomized embedded scheduler, we are interested in 
whether every thread progresses with probability 1 [dAFM ROS] . Such a qualitative question 
is much easier to study, and to justify, than its quantitative version; indeed, if we asked 
for a lower bound p < 1 for the probability of progress, the choice of p would need to be 
justified by an analysis of how much failure probability is acceptable in the final system, 
an analysis that is generally not easy to accomplish. For the same reason, the correct- 
ness of randomized distributed algorithms is often established with respect to qualitative, 
rather than quantitative, criteria (see, e.g., [PSLOOl iKNPOOl ISto02] ). Furthermore, since 
qualitative answers can generally be computed more efficiently than quantitative ones, they 
are often used as a useful pre-processing step. For instance, when computing the maximal 
probability of reaching a set of target states T, it is convenient to first pre-compute the 
set of states Ti D T that can reach T with probability 1, and then compute the maximal 
probability of reaching T: this reduces the number of states where the quantitative question 
needs to be answered, and leads to more efficient algorithms jdAKN"'"00] . Lastly, we remark 
that qualitative answers, unlike quantitative ones, are more robust to perturbations in the 
numerical values of transition probabilities in the MDP. Thus, whenever a system can be 
modeled only within some approximation, qualitative verification questions yield informa- 
tion about the system that is more robust with respect to modeling errors, and in many 
ways, more basic in nature. 

In this paper, we provide logics for the specification of qualitative properties of Markov 
decision processes, along with model-checking algorithms for such logics, and we study the 
equivalence relations arising from such logics. Our starting point for the logics is provided by 
the probabilistic logics pCtl and pCtl* [HJ9l I ASB+951 IBdA95j . These lo gics are able to 



express bounds on the probability of events: the logic pCtl is derived from Ctl by adding 
to its path quantifiers V ("for all paths") and 3 ("for at least one path") a probabilistic 
quantifier P. For a bound q € [0, 1], an inequality tx\£ {<, <, >, >}, and a path formula ip, 
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the pCtl formula P|>ag(/5 holds at a state if the path formula (p holds from that state with 
probability ixi q. The logic pCtl* is similarly derived from Ctl*. In order to obtain logics 
for qualitative properties, we consider the subsets of pCtl and pCtl* where V, 3 have 
been dropped, and where the bound q against which probabilities are compared can assume 
only the two values 0, 1. We call the resulting logics Qrctl and Qrctl*, for Qualitative 
Randomized Ctl and Ctl*. 

We provide symbolic model-checking algorithms for the logic Qrctl; these algorithms 
can be easily extended to Qrctl*, since for MDPs the verification of general temporal-logic 
properties can be reduced to reachability questions jCYQSj [dA97aj . As usual, the model- 
checking algorithms for Qrctl proceed by induction on the structure of a formula. The 
cases for some of the operators are known; for others, we give new algorithms, completing 
the picture of the symbolic algorithms required for Qrctl model checking. 

We then proceed to study the equivalence relations that arise from Qrctl. For two 
states s and t of an MDP, we write s t if the states s, t satisfy the same Qrctl 
formulas; similarly, Qrctl* induces the relation . Informally, s t holds if the set of 
properties that hold with probability 0, positive, and 1, at s and t coincide. These relations 
are thus strictly coarser than standard probabilistic bisimulation [SL94j , which relates states 
only when the precise probability values coincide. Other works ( |DGJP99] ) have introduced 
distances which quantify the difference in the probabilistic behavior of two MDPs. When 
the distance between s and t is zero, s and t are probabilistically bisimilar, and so they are 
also qualitatively bisimilar. Aside from that, the distance between two states is in general 
unrelated to the states being qualitatively equivalent or not. 

The appeal of the relations kt'^ and lies in their ability to relate implementations 
and specifications in a qualitative way, abstracting away from precise probability values. 
The relations, and their asymmetrical counterparts related to simulation, are particularly 
well-suited to the study of refinement and implementation of randomized algorithms, where 
the properties to be preserved are most often probability-1 properties. For instance, when 
implementing a randomized thread scheduler [dAFMROSj . the implementation needs to 
guarantee that each thread is scheduled infinitely often with probability 1; it is not im- 
portant that the implementation realizes exactly the same probability of scheduling each 
thread as the specification. Our qualitative relations can also be used as a help to analyze 
qualitative properties of systems, similarly to how bisimulation reductions can help in veri- 
fication. Given a system, the relations enable the construction of a minimized, qualitatively 
equivalent system, on which all qualitative questions about the original system can be an- 
swered. We will show that our qualitative equivalences are computable by efficient discrete 
graph-theoretic algorithms that do not refer to numerical computation. 

We distinguish between alternating MDPs, where probabilistic and nondeterministic 
choices occur at different states, from the general case of non- alternating MDPs, where both 
choices can occur at the same state. Our first result is that on finite, alternating MDPs, the 
relation coincides with alternating bisimulation [AHKV98] on the MDP regarded as 
a two-player game of probability vs. nondeterminism. This result enables the computation 
of K,^^ via the efficient partition-refinement algorithms developed for alternating bisimula- 
tion. We show that the correspondence between and alternating bisimulation breaks 
down both for infinite MDPs, and for finite, but non-alternating, MDPs. Indeed, we show 
that on non-alternating MDPs, the relation k,^^ cannot be computed by any partition- 
refinement algorithm that is local, in the sense that partitions are refined by looking only 
at 1-neighbourhoods of states (the classical partition-refinement algorithms for simulation 



4 



K. CHATTERJEE, L.DE ALFARO, M. FAELLA, AND A. LEGAY 



and bisimulation are local). These results are surprising. One is tempted to consider al- 
ternating and non-alternating MDPs as equivalent, since a non-alternating MDP can be 
translated into an alternating one by splitting its states into multiple alternating ones. The 
difference between the alternating and non-alternating models was already noted in [ST05j 
for strong and weak "precise" simulation, and in [BSOlj for axiomatizations. Our results 
indicate that the difference between the alternating and non-alternating model is even more 
marked for Ri'''^, which is a local relation on alternating models, and a non-local relation in 
non-alternating ones. 

More surprises follow when examining the roles of the Q ("next") and lA ("until") 
operators, and the distinction between Qrctl and Qrctl*. For Ctl, it is known that 
the O operator alone suffices to characterize bisimulation; the li operator does not add 
distinguishing power. The same is true for Qrctl on finite, alternating MDPs. On the 
other hand, we show that for non-alternating, or infinite, MDPs, lA adds distinguishing 
power to the logic. Similarly, the relations induced by Qrctl and Qrctl* coincide on 
finite, alternating MDPs, but Qrctl* has greater distinguishing power, and induces thus 
finer relations, on non-alternating or infinite MDPs. 

In summary, we establish that on finite, alternating MDPs, qualitative equivalence 
can be computed efficiently, and enjoys many canonical properties. We also show that the 
situation becomes more complex as soon as infinite or non-alternating MDPs are considered. 
In all cases, we provide sharp boundaries for the classes of MDPs on which our statements 
apply, distinguishing also between finitely and infinitely-branching MDPs. Our results also 
indicate how the distinction between alternating and non-alternating MDPs, while often 
overlooked, is in fact of great importance where the logical properties of the MDPs are 
concerned. 

Our organization of the paper is as follows: in section[2]we present the formal definitions 
of MDPs and the logics Qrctl* and Qrctl. In section [3] we present a model checking 
algorithm for MDPs with the logic Qrctl. In section [4] we characterize the equivalence 
relations of MDPs with respect to Qrctl. In section [5] we present algorithms to compute 
the equivalence relations. Finally, in section [6] we discuss the roles of the until and wait-for 
operators in the logics, and in section [7| we consider the role of linear-time nesting (i.e., the 
equivalences for the logic Qrctl*). 

2. Definitions 

2.1. Markov Decision Processes 

A probability distribution on a countable set X is a function f : X ^ [0, 1] such that 
Ylix&x /(^) ~ denote the set of all probability distributions on X by ^{X). Given 

/ G V{X), we define Supp{f) = {x & X \ f{x) > 0} to be the support of /. We consider 
a fixed set AP of atomic propositions, which includes the distinguished proposition turn. 
Given a set S, we denote (respectively S'^) the set of finite (resp. infinite) sequences of 
elements of S. 

A Markov decision process (MDP) G = {S,A,T,6, [■]) consists of the following compo- 
nents: 

• a countable set of states S; 

• a finite set of actions A; 
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• an action assignment F : 5 i-^ 2 \ 0, which associates with each state s G the set T{s) 
of actions that can be chosen at s; 

• a transition function 5 : S x A ^("S*), which associates with each state s and action a 
a next-state probabihty distribution 6{s,a); 

• a labehng function [•] : S i-^ 2^^, which labels all s G with the set [s] of atomic 
propositions true at s. 

For s € S and a € F(s), we let Dest{s,a) = Supp{5{s,a)) be the set of possible destinations 
when the action a is chosen at the state s. The MDP G is finite if the state space S 
is finite, and it is finitely-branching if for all s G /S and a G F(s), the set Dest{s,a) is 
finite. A play or path is an infinite sequence uj = (sq, si, . . .) G of states of the MDP. 
For s G S and q G AP, we say that s is a ^-state iff g G [s]. We define an edge relation 
E = {{s,t) e S X S \ 3a e r(s) .t G Dest{s,a)}; for s G 5, we let E{s) = {t \ {s,t) G E}. 
An MDP G is a Markov chain if |F(s)| = 1 for all s G 5; in this case, for all s,t G S" we 
write 6{s){t) rather than 6{s,a){t) for the unique a G F(s). 

Interpretations. We interpret an MDP in two distinct ways: as a lY2-player game, and as 
a 2-player game. In the lY2-player interpretation, probabilistic choice is resolved proba- 
bilistically: at a state s G 5", player 1 chooses an action a G F(s), and the MDP moves 
to the successor state t G S with probability 5{s,a){t). In the 2-player interpretation, we 
regard probabilistic choice as adversarial, and we treat the MDP as a game between player 1 
and player p (p for "probability"): at a state s, player 1 chooses an action a G F(s), and 
player p chooses a destination t G Dest{s,a). The lY2-player interpretation is the classical 
one |Der70] . The 2-player interpretation will be used to relate the qualitative equivalence 
relations for the MDP with the alternating relations of |AHKV98] . and thereby derive al- 
gorithms for computing the qualitative equivalence relations. 

Strategies. A player-1 strategy is a function a : ^(^) that prescribes the probability 

distribution a{'w) over actions to be played, given the past sequence w G of states visited 
in the play. We require that if a G Supp{a{w ■ s)), then o G F(s) for all o G ^, s G S, and 
w G S*. We denote by S the set of all player-1 strategies. 

A player-p strategy is a function vr : S"*" x A '^{S)- The strategy must be such that, 
for all s G S", u; G S* , and a G F(s), we have that Supp{Tr{'w ■ s,a)) C Supp{5{s, a)). Player p 
follows the strategy vr if, whenever player 1 chooses move a after a history of play w, she 
chooses the destination state with probability distribution 7r(t<;, a). Thus, in the 2-player 
interpretation, nondeterminism plays first, and probability second. We denote by 11 the set 
of all player-p strategies. 

The 2-player interpretation. In the 2-player interpretation, once a starting state s € S and 
two strategies cr G S and vr G Ft have been chosen, the game is reduced to an ordinary 
stochastic process, and it is possible to define the probabilities of events, where an event 
^ C 5''^ is a measurable set of paths. We denote the probability of event A, starting 
from s G S*, under strategies a G S and vr G 11 by Pr^''^(^): note that the probability of 
events given strategies a and vr do not depend on the transition probabilities of the MDP 
as the strategy vr can chose any probability distribution at each step. Given s G S" and 
(T G S, vr G n, a play (sq, si, . . .) is feasible if for every /c G N, there is a G F(sfc) such that 
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(t(so, Si, ... , Sk){a) > and 7r(so, si, . . . , ■Sk,a){.Sk+i) > 0. We denote by Outc(s, a, vr) C 5"^ 
the set of feasible plays that start from s given strategies a and vr. 

The 1^/2-player interpretation. In the lY2-player interpretation, we fix for player p the 
strategy vr* that chooses the next state with the distribution prescribed by 5. Precisely, for 
aU w e S*, s e S, and a G T{s), we let ■k*{w ■ s,a) = 6{s,a). We then write Pr^(>i) and 
Outc(s,o") instead of Pr^''^ (A) and Outc (s, cr, vr*), respectively, to underline the fact that 
these probabilities and set of outcomes are functions only of the initial state and of the 
strategy of player 1. 

Alternating MDPs. An alternating MDP (AMDP) is an MDP G = {S, A,T ,5,[]) along 
with a partition [Si, Sp) of S such that: 

(1) If s G Si, then turn G [s\ and, for all a G T{s), \Dest{s,a)\ = 1. 

(2) If s G Sp, then turn [s] and |r(s)| = 1. 

The states in Si are the player-1, or nondeterministic states, and the states in Sp are 
the player-p, or probabilistic states. The predicate turn ensures that the MDP is visibly 
alternating: the difference between player-1 and player-p states is obvious to the players, 
and we want it to be obvious to the logic too. Alternating MDPs can be represented 
more succinctly (and more intuitively) by providing, along with the partition (Si,Sp) of 
S, the edge relation E C S x S, and a probabilistic transition function 5 : Sp 1— > T>{S). 
The probabilistic transition function is defined, for s G Sp, t G S, and a G T{s), by 
S{s){t) = 6{s,a){t). A non- alternating MDP is a general (alternating or not) MDP. 

We represent MDPs by graphs: vertices correspond to nodes, and each action a from a 
state s is drawn as a hyperedge from s to Dest{s,a). 

2.2. Logics 

We consider two logics for the specification of MDP properties. The first, Qrctl*, is a 
logic t hat captu res qualitative properties of MDPs, and is a qualitative version of pCtl* 
[HJ941 lASB+951 IBdA95| . The lo gic is defined with respect to the classical, lY2-player 
semantics of MDPs. The second logic, Atl*, is a game logic defined with respect to the 
2-player semantics of MDPs as in |AHK02) . 

Syntax. The syntax of both logics is given by defining the set of path formulas ((/?) and state 
formulas (ip) via the following inductive clauses: 



where q G AP is an atomic proposition, tt is the boolean constant with value true, and PQ 
is a path quantifier. The operators U, W and Q temporal operators. The logics Atl* 
and Qrctl* differ in the path quantifiers: 



• The path quantifiers in Qrctl* are: 3'^", V''", 3'*°™^ V'*''™^ 3YvY3>° and V>°. 

• The path quantifiers in Atl* are: ((!)), {{p)), {{l,p)), {{$)). 



path formulas: if 
state formulas: ip 



tt I g I ^V- 1 V V V I PQif ); 
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The fragments Atl of Atl* and Qrctl of Qrctl* consist of formulas where every temporal 
operator is immediately preceded by a path quantifier. In the following, when we refer to 
a "formula" of a logic, without specifying whether it is a state or path formula, we always 
mean a state formula. As usual, we define Bip and (}ip to be abbreviations for (pW{-itt) 
and ttU(p, respectively. 



Semantics. For a play = {sq, si, . . .) we denote by Ld[i] the play starting from the i-th state 
of uj, i.e., Lj[i] = {si, Si+i, . . .). The semantics for the path formulas is defined as follows, for 
path formulas (p, ipi, ip2' 

uj \= iS to ^ if 

Lo \= ipiU(p2 iff 3j G N.d;[j] |= ip2 and VO < i < j. Ld[i] \= ipi 

Lo \= ipiyVip2 iff (Vj G N. \= (fii) or 3j G N. \= (p2 and VO < i < j. uj[i] \= (pi. 
Observe that 

-^{ipiUip2) = □(-V'2) V (-V2W(-V'i A = -■V'2W^Vi- 

Finally, we have 

Co \= Tp So \= ij: . 

Given a path formula (p we denote by {(p} = {to \ Co \= (p} the set of plays that satisfy (p. 
The semantics of the state formulas of Atl* and Qrctl* is defined as follows, for a state 
s, path formula ip, and state formulas tpi and 'ip2' 

s\=tt 

s \= q iff g' G [s] 

s \= -I'i/'i iff s ^ ■i/'i 

s 1= -01 V "02 iS s \= tpi or s \= 11)2 

s \= 3'^"(^) iff 3(7 G S. Outc(s, a) C {pj 
s \= V«"((^) iff V(7 G S. Outc(s, a) C yj 

s^3'ip) iff3aGS.Pr^(M) = l 
s\=y'{^) iffVaGS. Pr-(M) = 1 

s \= 3>0(^) iff 3a G S. Pr^dv?]) > 
s ^ V>0((^) iff Vc7 G S. Pr^(y ) > 

s \= 3'°'^%ip) iff 3(7 G S. Outc(s, a) n / 
s H V''°™^ {(p) iff Vc7 G S. Outc(s, a) n |^] 7^ 

s ^ {{l)){p) iff 3cr G S.Vvr G n.Outc(.s, ct, vr) C [i^] 

s ^ {{p)) {p) iff 3^ G n.V(T G S.Outc(s, cr, ^) C {pj 

s ^ iff 3c7 G S.37r G n.Outc(s, a, tt) C ^ 

s \= ((0)) (ip) iff V(7 G S.Vtt G n.Outc(s, a, tt) C |(^| . 

Given an Atl* or Qrctl* formula (p and an MDP G = {S, A,T,S,[-]), we denote by 
bPlc = ^ "S* I s t= (^} the set of states that satisfy the state formula ip, and we omit 
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Figure 1: A simple Markov chain. 

the subscript G when obvious from the context. For ah path formulas (/? of Qrctl, the 
following dualities hold: 

P'^Vl = h(v^''"'^(-^))l 
p-™vi = h(v'^"(-¥'))l 
p>Vl = h(v^(-^))l 
PVI = h(v>°h^))l. 

We now present a simple example to illustrate the difference between the satisfaction of a 
path formula with probability 1 and for all paths. 

Example 2.1. Consider the simple Markov chain shown in Figure [TJ Let the propositions 
true at states s and the q and r, respectively. Let us consider the starting state as s, and the 
formula '(^r (eventually r). The formula holds at state s with probability 1, since the only 
closed recurrent set of states in the Markov chain is the state t (labeled with proposition r). 
Hence holds in state s with probability 1. However, there is a path (namely, s^) that 
violates the property eventually r, but the probability measure for the set {s^} of paths 
is 0. Thus the state s does not satisfy that all on all paths we have eventually r, though it 
satisfies the property eventually r with probability 1. If we consider the property eventually 
q, then for all paths starting from s the property holds (hence the property also holds with 
probability 1). 

The following lemma establishes a relationship between Qrctl and Atl, proving that 
the Qrctl quantifiers with superscript all and some are equivalent to the Atl quantifiers. 

Lemma 2.2. For all path formulas ip, the following equivalences hold. 

l{{i)M = p'^'Vl 

m,p)M = p^°™vi 
imM = [v^vi 

Proof. Let G = {S,A,T,S, [■]) be an MDP and let s G 5. We prove the first statement. 
Assume s \= {{l))(p. By definition, there exists a* £ T, such that: 

Vvr G n.0utc(s,cj*,7r) C yj. 

Let vr* G H be the strategy of player p that chooses the next state according to 5 (i.e., the 
natural strategy of player p in G). We have: 

Outc(s,cj*) = Outc(s,c7*,7r*) C yj. (2.2) 

Therefore, s \= 3"''(/;. 

Conversely, assume s \= 3'^^^(p. Then, there exists o"* G S such that (j2.2p holds. Let vr 
be any strategy of player p. We have that Outc(s, a*, tt) C Outc(s, cj*, vr*), because vr* is 
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the most liberal strategy for player p, i.e., no player-p strategy can ever choose a successor 
state that is not among those that are chosen by vr*. Therefore, Outc(s, a*, tt) C [i^] and 

s N ((1))^- 

Next, we prove the second statement. The remaining statements follow by duality. 
Assume s \= {{l,p))(p. Then, there exist cr* G S and vr* G 11 such that Outc(s, cr*, vr*) C \ip\. 
Let TT* be the natural strategy for player p in G. By the previous argument, Outc(s, a' , tt') C 
Outc(s,(7',7r*). Therefore, Outc(s, a', vr*) n |</?] / and s ^ B^"™'^^;. 

Finally, assume s \= 3'^°"^^ip. By definition, there exists fi* G S such that Outc(s, ci*, vr*) n 
[93] 7^ 0, where vr* is the natural strategy for player p in G. Let a; be a play in Outc(s, a*, vr*)n 
[99] . Define a' and vr* as the deterministic strategies that give as only outcome uj. We have: 

Outc(s,cT*,vr*) = {a;} C 

Therefore, s \= {{l,p))ip. I 

Finally, the following lemma proves the equivalence of some Qrctl formulas. 
Lemma 2.3. For all atomic propositions q,r, and for all MDPs, we have: 

p>0O9l = P^''"'^Ogl 
9l = P"" O'zl 

p>°g^/r] = l3'°'^''qUr\ (2.3) 
p^gWr] = p""g>Vr]. 

Proof. The first two statements are obvious by definition. The third statement follows by 
noting that s \= 3^°™^qUr iff there is a finite path in {S,E) from s to an r-state, and all 
states of the path, except possibly the last, are g-states. If such a path exists, there is 
certainly a strategy of player 1 that follows it with positive probability. 

For the last statement, the "5" inclusion is obvious by definition. For the other in- 
clusion, assume by contradiction that s G [B^gWr], but all strategies of player 1 ensuring 
qWr with probability one also exhibit a path violating it. Then, s G ||V'"'™'^ ~'(^Wr)] = 
|ysome Following an argument similar to the one for the third statement, we obtain 

that s G IV>° ^rU^qj = |V>° -'(gWr)], which is a contradiction. I 



2.3. Equivalence Relations 

Given an MDP G = {S,A,T,6, [•]), we consider the equivalence relations induced over its 
state space by various syntactic subsets of the logics Qrctl and Atl. Define the following 
fragments of Qrctl: 

• Qrctl^*^ is the syntactic fragment of Qrctl containing only the path quantifiers 3^" 
and V>0; 

• Qrctl*^" is the syntactic fragment of Qrctl containing only the path quantifiers 3"" 
and V''". 

Note that, because of the dualities (j2.ip . we do not need to consider the fragments for V^, 
^1 ^ ysome^ ^some _ rpj^^ relations induced by Qrctl>° and Qrctl*^" provide us with a notion 
of qualitative equivalence between states. 

«>o = {(s, s')eSxS I VV' G Qrctl>°, s ^ -0 iff s' N V'} 
= {(s, s')eSxS I VV' G Qrctl'^", s^i^iSs' ^ ^}. 
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Game 



finite 




Figure 2: Relationship between equivalence relations for AMDPs. 

We denote by f«>'^'0 the equivalence relation defined by Qrctl^*^, with Q a.s the only 
temporal operator. We also define the equivalences and as the QRCTL*-version of 
ss^*^ and ss"", respectively. 

The syntactic subset of Atl which uses only the path quantifiers and ((0)) induces 

the usual notion of bisimulation |Mil90j : indeed, quantifiers {{l,p)) and ((0)) correspond to 
quantifiers 3 and V of Ctl |CE81j . respectively. The syntactic subset of Atl which uses 
only the path quantifiers ((!)) and {{p)) induces alternating bisimulation |AHKV98] . We 
have: 

~TS = {(■5, s') S X S \ for all Atl formulas ip with 

{{l,p)), ((0)) as path quantifiers, s \= ip iE s' \= ip}; 

~Game = {(s, s') & S X S \ for all Atl formulas ip with 
((1)), {{p)) as path quantifiers, s \= ip iS s' \= ip}; 

~ATL = {(s, s') G S X S \ for all Atl formulas ip, s \= ip iE s' \= Tp}; 

where TS is the short form for transition systems. In the relation ~Gamei nondeterministic 
and probabilistic choice represent the two players of a game. In the relation ~tS) nonde- 
terminism and probability always cooperate as a single player. Finally, the relation ~atl 
arises from the full logic Atl, where nondeterminism and probability can be either antago- 
nistic or cooperative. The relations ~tS) ~Game, and ~atl can be computed in polynomial 
time via well-known partition-refinement algorithms [Mil90[ IAHKV98] . 

Figure [2] (resp. Figure [3]) summarizes the relationships between different equivalence 
relations on alternating MDPs (resp. general MDPs) that we will show in this paper. An 
arrow from relation A to relation B indicates that A implies B, i.e., that A is finer than B. 



3. Model Checking Qrctl 

In order to characterize the equivalence relations for Qrctl, it is useful to present first 
the algorithms for Qrctl model checking. The algorithms are based on the results of 
[dA97al ldA97bl IdAHOO] : see also |CdAH04j . As usual, we present only the algorithms 
for formulas containing one path quantifier, as nested formulas can be model-checked by 



QUALITATIVE LOGICS AND EQUIVALENCES FOR PROBABILISTIC SYSTEMS 



11 




Figure 3: Relationship between equivalence relations for MDPs. 



recursively iterating the algorithms. As a consequence of dualities (j2.ip . we need to provide 
algorithms only for the operators 3 Q), 3U, and 3VV, and for the modalities all, i , > 0, and 
some. The algorithms use the following predecessor operators, for X,Y C S: 

Pre{X) = {s£S\3ae r(s) . Dest{s, a) n X / 0} 

Cpre{X) = {seS\3ae r(s) . Dest{s,a) C X} 

Apre{Y, X) = {seS\3ae r(s) . Dest{s, a) C y A Dest{s, a) n X ^ 0}. 

The operators Pre and Cpre are classical; the operator Apre is from |dAHK98] . We write 
the algorithms in //-calculus notation [Koz83] . Given an MDP G = {S, A,T,6,[-]), the 
interpretation ItpJ of a /i-calculus formula ip is a subset of states. In particular, for a 
propositional symbol q G AP, we have [9] = {s € 5" | g G [s]} and [-ig] = {s € S" | 
q [s]}. The operators U, fl, and the above predecessor operators are interpreted as the 
corresponding operations on sets of states, and // and u indicate the least and greatest 
fixpoint, respectively. The following result directly leads to model-checking algorithms for 
Qrctl. 

Theorem 3.1. For atomic propositions q and r, and for all MDPs, the following equalities 
hold: 

13' Oq\ = [3'''' Oq} = Cpre{lqY) (3.1) 
P>0 Oq\ = P^"™"^ O ^1 = PreiM) (3.2) 
l3^\Ur\ = /iX([rl U {{q} D Cpre{X))) (3.3) 
l3>%Ur\ = p-™^gZYrl = /.X([rl U (M n Pre{X))) (3.4) 
p""g>Vrl = p^gWrl = z^y.(|rl U ([g] n Cpre{Y))) (3.5) 
p.ome^y^^j ^ ^Y.{lrj U ([g] n Pre(y))) (3.6) 
If the MDP is finite, the following equalities also hold: 

l3'qUrj = uY.fiX.ilrj U {{qj D Apre{Y,X))) (3.7) 
p>Og>Vr] = l3>^qU{{r A g) V 3""ng)]. (3.8) 
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Proof. The formulas involving the all and some modalities (i.e., statements (13. ip to (13. 6p ) 
are derived by the corresponding classical game algorithms, thanks to Lemma 12.21 and 
Lemma 12.31 Formula (j3.7p is from ^dAHK98j. Formula (j3.8p can be understood as follows. 
A closed component is a subset of states T C S such that, for all s G T, there is at least one 
a G r(s) such that Dest{s,a) C T. Using the relation qWr = {qU{r A q)) V Uq |MP91| . we 
have for s G S that s |= 3>°gWr \S. (i) s\= 3^^qU{q Ar), or (^iij there is a closed component 
T composed only of g-states, and a path sq, si, . . . , s„ in {S, E) composed of g-states, with 
So = s and Sn & T (see, e.g., |dA97a] ). Formula (j3.8p encodes the disjunction of (^ij and 
(n). I 

Note that, even though (j3.8p is not a //-calculus formula, it can be readily translated into 
the /i-calculus via (j3.4p and (|3.5p . Also observe the //-calculus formulas corresponding to 
Qrctl are either alternation free or contain one quantifier alternation between the fi and 
operator. Thus, from the complexity of evaluating //-calculus formulas we obtain the 
following result. 

Theorem 3.2. Given a finite MDP G = {S,A,T,5, [•]) and a Qrctl formula ip, the set 
{iPJq can be computed in 0{\S\ ■ \d\ ■ i) time, where \5\ = l^^gg X]aGr(s) \Dest{s, a)\ and £ 
denotes the length ofip. 

Proof. We first consider the computation of Pre(X), Gpre{X), and Apre{Y, X) for X,Y CI 
S. To decide whether s G Pre{X) we check if there exists a G T{s) such that Dest{s, a)nX ^ 
0. Similarly, to decide whether s G Cpre{X) (resp. Apre{Y,X)) we check if there exists 
a G r(s) such that Dest{s,a) C X (resp. Dest{s,a) C Y and Dest{s, a) D X 7^ 0). It follows 
that given sets X and Y, the sets Pre{X), Gpre{X), and Apre{Y,X) can be computed 
in time 0(^^g_5 ^^^^ |-Desi(s, a)|). Given a formula ip in Qrctl, with all of its sub- 
formulas already evaluated, it follows from Theorem 13.11 that the computation of [^] can be 
obtained by computing a //-calculus formula of constant length with at most one quantifier 
alternation of // and ly. Using the monotonicity property of Pre, Cpre and Apre, and the 
computation of Pre, Cpre and Apre, it follows that each inner iteration of the //-calculus 
formula can be computed in time Oi^^^^J^aeA \Dest{s,a)\). Since the outer iteration of 
the /t-calculus formula converges in l^l iterations, it follows that {ipj can be computed in 
time 0(|S'|-X]s£s SaeA \Dest{s, a)\). By a bottom-up algorithm that evaluates sub-formulas 
of a formula first, we obtain the desired bound for the algorithm. I 



4. Relationship between Qrctl and Atl Equivalences 

In this section, we compare the relations induced by Qrctl and Atl. These comparisons 
will then be used in Section [5] to derive algorithms to compute and ps^*^. 

We first compare Ri"^" with the relations induced by Atl. As a first result, we show 
that the relations induced by Atl coincide on alternating MDPs (AMDPs). This result 
follows from the fact that the turn is visible to the logic. 

Proposition 4.1. On AMDPs, we have ~Game = ^TS- 

Proof. Since the turn is observable (via the truth- value of the predicate turn), both ~Game 
and ~TS can relate only states where the same player (1 or p) can choose the next move. 
Based on this observation, the equality of the relations can be proved straightforwardly by 
induction. I 
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Figure 4: An infinite Markov chain in which states s and s' cannot be distinguished by 
Qrctl^'^, but are distinguished by the Atl formula {{p))0^q. 

Corollary 4.2. On AMDPs, we have ^atl = ~Game = ^TS- 

An immediate consequence of Lemma 12.21 is that and ~atl coincide. This enables 
the computation of via the algorithms for alternating bisimulation [ AHKV98] . 

Proposition 4.3. For all MDPs, Ri''" = ^atl- 

Next, we examine the relationship between and ~atl- On finitely-branching 
MDPs, is finer than ~atl; the result cannot be extended to infinitely-branching MDPs. 

Theorem 4.4. The following assertions hold: 

(1) On finitely-branching MDPs we have C ss^ir^. 

(2) There is an infinitely-branching AMDP on which ss^*^ ^ ~ATL- 

Proof. Assertion 1. For n > 0, we consider the n-step approximation ^^-pL ~atl- In 
finite MDPs, we have ~atl=~atl ~ \^\'-> finitely-branching MDPs, we have ~atl= 
nJ^Q ~atL' ^^^^ does not extend to MDPs that are not finitely-branching. We define a 
sequence ^'o, ^'i, ^I'2) • • • of sets of formulas such that, for all s,t € S, we have s ~atl ^ iff 
and t satisfy the same formulas in To this end, given a finite set ^ of formulas, we denote 
by BoolC(\I') the set of all formulas that consist in disjunctions of conjunctions of formulas 
in {ip, -^ip I ijj G ^'}. We assume that each conjunction (resp. disjunction) in BoolC(^') does 
not contain repeated elements, so that from the finiteness of ^ follows the one of BoolC(^'). 
We let ^0 = BoolC(^P) and, for > 0, we let ^^+1 = BoolC(^fc U {3>o Q V', 3"" Q ^ I 
■0 G ^k})- The formulas in BoolC(^'o), BoolC(^'i), . . . ,BoolC(^'„) provide witnesses that 
ps^^ C ~ATL- Thus for all n, we have ps^^ C and it follows that k,^^ C. «atl- 

Assertion 2. Consider a Markov chain, depicted in FigureUl with state space S = NU{s, s'}, 
with only one predicate symbol q, such that [0] = {g}, and [t] = for all t G 5 \ {0}. There 
is a transition from s to every i G N with probability 1/2*"^^. There is a transition from s' 
to s' with probability 1/2, and from s' to every i G N with probability 1/2*"*"^. There is a 
transition from i G N with i > to every state in {j G N | j < i}, with uniform probability. 
There is a deterministic transition from to itself. Since this is a Markov chain, the two 
path quantifiers 3 and V are equivalent, and we need only consider formulas of the form 
3^*^ and 3^. By induction on the length of a Qrctl formula ^p, we can then show that (p 
cannot distinguish between states in the set {i G N [ i > U {s, s'}. Hence, s s'. On 
the other hand, we have s ^atl s', since s ^ and s' \= {{p))0^q. I 
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To obtain a partial converse of this theorem, we need to translate all Qrctl formulas into 
Atl. For finite MDPs, Lemmas 12.21 and 12.31 enable us to translate all Qrctl formulas, 
except for formulas of the type 3^ U and For the latter type, from (j3.8p together 

with Lemmas 12.21 and 12.31 we obtain the following result. 

Lemma 4.5. For finite MDPs, and for all atomic propositions q, r, we have 

[3>%Wr\ = l{{l,p)){qU{{qAr) V {{l))aq))l 

Regarding formulas of the type 3^ lA, they can be model-checked using the /x-calculus ex- 
pression (j3.7p . To obtain a translation into Atl, which will be given in proof of Theorem l4.7^ 
we first translate into Atl the operator Apre. To this end, for Atl formulas 93, V) define 

Lemma 4.6. For AMDPs, and for all Atl formulas (p, ^jj, we have [-F^pr-eCVi "0)1 = 
Apre{M,m. 

Proof. We consider the following characterization of the Apre operator, valid for AMDPs: 
for sets X and y, and a state s we have s G Apre{Y, X) iff the following conditions hold: 
(a) if s G Si, then there exists a G r(s) such that 6{s, a) X OY; and (b) if s G Sp, then for 
the unique action a G r(s), we have Dest{s, a) (^Y and Dest{s, a) H A" 7^ 0. The definition 
of 

FApre captures the above two conditions. The result follows. I 

Note that the lemma holds only for alternating MDPs: indeed, we will show that, on 
non-alternating MDPs, the operator Apre is not translatable into Atl. 

Using these lemmas, we can show that on finite AMDPs, we have ~atl ^ This 
result is tight: we cannot relax the assumption that the MDP is finite, nor the assumption 
that it is alternating. 

Theorem 4.7. The following assertions hold: 

(1) On finite AMDPs, we have ^atl Q 

(2) There is a finite MDP on which ^atl 2 

(3) There is an infinite, but finitely-branching, AMDP on which ^atl 

Proof. Assertion 1. We prove that on a finite, alternating MDP, the counterpositive holds: 
if s 96^^ t, then s 96 atl t- Let s and t be two states such that s 96^*^ t. Then, there must 
be a formula ip in Qrctl^'^ that distinguishes s from t. From this formula, we derive a 
formula /(v?) in Atl that distinguishes s from t. 

We proceed by structural induction on ip, starting from the inner part of the formula 
and replacing successive parts that are in the scope of a path quantifier by their Atl version. 
The cases where p is an atomic proposition, or a boolean combination of formulas are trivial. 
Using ()2.ip . we reduce QRCTL^'^-formulas that involve a V operator to formulas that only 
involve the 3 operator. Lemma [2.31 provides translations for all such formulas, except those 
of type 3^(v9^V)- For instance, ([23]) leads to f{3>°ipUi;) = {{l,p))f{ip)Uf{'ip). In order 
to translate a formula of the form 7 = 3^ (ipUip), we translate the evaluation of the nested 
//-calculus formula (j3.7p into the evaluation of a nested Atl formula, as follows. Define the 
set of formulas {onj | < i,j < n}, where n = \S\ is the number of states of the AMDP, 



2 f«>0. 
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Figure 5: States s and t cannot be distinguished by Atl, but are distinguished by B-'Oq. 




g q g g 

Figure 6: An infinite Markov chain on which ~atl 2 where Xj's and y^'s represent 

the probabihties that the corresponding edge is taken. 

via the following clauses: 

Vi G [0..n] : aifl = fF 
Vj € [l..n] : aoj = it 
yi € [l..n] . Vj € [0..n - 1] : 

«i,j+l = filp) V {f{ip)AFApreioii-l,n,ai,j))- 

From Lemma 14.61 the above set of formulas encodes the iterative evaluation of the nested 
fixpoint (j3.7p . so that we have |an,nl = ll}, and we can define f{'~f) = an,n- This concludes 
the translation. 

Assertion 2. Consider the MDP shown in Figure [5l The states s and t are such that 
(s,i) G~ATL- However, s \= 3^(Og) (consider the strategy that plays always a), whereas 

Assertion 3. Consider the infinite AMDP shown in Figure [6l All states are probabilistic 
states, i.e. Si = 0. For all i > 0, we set Xj = ^ a^^d yi = 2 2» ^ so that ni>o = and 
Wi>Q yi ~ \- i\iaX s ~ATL t- However, s \= 3^^{Uq) and t ^ 3^^{Oq). I 

The example in Figure[5]also shows that on non-alternating MDPs, unlike on alternating 
ones (see Lemma l4.6p . the Apre operator cannot be encoded in Atl. If we were able to 
encode Apre in Atl, by proceeding as in the proof of the first assertion, given two states s, 
t with s 96>o t, we could construct an Atl formula distinguishing s from t. 

As a corollary to Theorems 14.41 and 14.71 we have that on finite, alternating MDPs, 
the equivalences induced by Atl and Qrctl coincide. Thus the discrete graph theoretic 
algorithms to compute equivalences for Atl can be used to compute the Qrctl equivalences 
for finite AMDPs. 

Corollary 4.8. For finite AMDPs, we have = ^atl- 
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(a) A non-alternating MDP. (b) An alternating MDP. 

Figure 7: MDPs illustrating how separating nondeterministic and probabilistic choice does 
not help to compute Ki^^. 

5. Computing Qrctl Equivalences 

In this section, we take advantage of the results obtained in Section U] to derive algorithms 
to compute and for AMDPs. We also provide an algorithm to compute those 
relations on non-alternating MDPs. 

5.1. Alternating MDPs 

Corollary 14.81 immediatelv provides an algorithm for the computation of the Qrctl equiva- 
lences on AMDPs, via the computation of the Atl equivalences (interpreting nondetermin- 
ism and probability as the two players). In particular, the partition-refinement algorithms 
presented in |AHK02] can be directly applied to the problem. This yields the following 
result. 

Theorem 5.1. The two problems of computing sa^" and on finite AMDPs are PTIME- 
complete. 

Proof. Consider a turn-based game and consider the AMDP obtained from the game as- 
signing uniform transition probabilities to all out-going edges from a player 2 state. Then 
the 2-player game interpretation of the AMDP coincides with the original turn-based game. 
The result then follows from Corollary 14.81 and from the PTIME-completeness of ATL 
model checking and computing ~atl |AHK02j . I 



5.2. Non-Alternating MDPs 

For the general case of non- alternating MDPs, on the other hand, the situation is not 
nearly as simple. First, let us dispel the belief that, in order to compute f*^*^ on a non- 
alternating MDP, we can convert the MDP into an alternating one, compute ss^*^ via ~atl 
(using Corollary 14. 8p on the alternating one, and then somehow obtain sa^" on the original 
non-alternating MDP. The following example shows that this, in general, is not possible. 



Example 5.2. Consider the MDP depicted in Figure 7(a) , where the set of predicates is 
AP = {q,r}. We have s ^'^^ s' . Indeed, the only difference between s and s' is that at 
state s' the action c is available: since c is a convex combination of a and b, s and s' are 
probabilistically bisimilar in the sense of [SL94j . and thus also related by ps^'^. We transform 
this MDP into an alternating one by adding, for each state s and each a € r(s), a state 
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(s, a) which represents the decision of choosing a at s; the result is depicted in Figure [7(b) [ 
In this AMDP, however, the state (s',c) has no equivalent, as it satisfies both 3^" Q Q 
and 3^*^ Q ^- Therefore, on this AMDP we have s ^-^^ s', as witnessed by the formula 

^aii o((3>o O^) A(3>o Or)). 

As the example illustrates, the problem is that once nondeterminism and probability are 
separated into different states, the distinguishing power of ^-'^ increases, so that computing 
^ATL on the resulting alternating MDP does not help to compute on the original non- 
alternating one. 

Failure of local partition refinement. Simulation and bisimulation relations can be com- 
puted via partition refinement algorithms that consider, at each step, the 1 -neighbourhood 
of each state: that is, the set of states reachable from a given state in one step [Mil90j . 
We call such algorithms 1 -neighbourhood partition refinements. Here, we show a general re- 
sult: no 1-neighbourhood partition refinement algorithm exists for f*^*^ on non-alternating 
MDPs. 

We make this notion precise as follows. Consider an MDP G = {S, A, T, 6, [•]), together 
with an equivalence relation ~ on S. Intuitively, two states are 1-neighbourhood isomorphic 
up to ^ if their 1-step future looks identical, up to the equivalence ~. Formally, we say 
that two states s,t S are 1-neighbourhood isomorphic up to ~, written s ~ i, iff s ~ t, 
and if there is a bijection R between E{s) and E{t), and a bijection R between r(s) and 
T(t), which preserve ^ and the transition probabilities. Precisely, we require that: 

• if s' G E{s) and t' G E{t) with s' Rt' , then s' ~ t'; 

• if o G r(s) and b G T{t) with aRb, then for all s' G E{s) and t' G E{t) with s' Rt', we 
have 5{s,a){s') = 6{t,b){t'). 

Let Parts be the set of equivalence relations on S. A partition refinement operator f : 
Parts I— > Parts is an operator such that, for all ~ G PartS, we have /(~) is finer than ~. 
We say that a partition operator computes a relation ~ if we have « = lim„^oo /"(~pred)) 
where /" denotes n repeated applications of / and s ^pred t iff [s] = [t] . 

We say that a partition refinement operator / is 1-neighbourhood if it refines an equiva- 
lence relation ~ on the basis of the 1-neighbourhood of the states, treating in the same fash- 
ion states whose 1-neighbourhoods are isomorphic up to ~. Precisely, / is 1-neighbourhood 
if, for all ~ G PartS and for all s, s', t,t' G S with s ~ s', t ~ t', we have either 
(s, t), (s', t') G /(~), or {s,t), {s' ,t') ^ /(~)- We can now state the non-existence of 1- 
neighbourhood refinement operators for ss^*^ as follows. 




Figure 8: MDP showing the lack of 1-neighbourhood refinement operators. 
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Theorem 5.3. There is no 1 -neighbourhood partition refinement operator which computes 
on all MDPs. 

Proof. Consider the states si, S2, S3, S4 of the MDP depicted in Figure[8l and take ~ = ^pred- 
Let / be any 1-neighbourhood partition refinement operator. From si ~ S2 ~ ss ~ S4, we 
can see that S2 ^ S3 ^ S4. Let ~' = /(~). Considering the pairs (si,S2), (51,53), and 
(si, S4) in the definition of 1-neighbourhood partition refinement operator, we have that ~' 
satisfies one of the foUowing two cases: 

(1) si S2 and si S3 and si 9^' S4, 

(2) si ~' S2 and si ~' S3 and si ~' S4. 

In the first case, the partition refinement terminates with a relation ^" such that si S2- 
This is incorrect, since we can prove by induction on the length of Qrctl^'' formulas that 
no such formula distinguishes si from S2, so that si S2- In the second case, the partition 
refinement terminates with a relation ~" such that si ~" S3. This is also incorrect, since 
the formula 3^<^r is a witness to si S3. We conclude that a 1-neighbourhood partition 
refinement operator cannot compute I 

To give an algorithm for the computation of Ri-*'^, given two sets of states Ci and C2, 

let: 

[/(Ci, C2) = {C3 = (so, si, ...) \ 3j > . Sj e C2 andy < i < j . Si e Ci} 
EU\Ci,C2) = {seS\3a£^. Pr^(C/(Ci, C2)) = !}• 

Intuitively, if Ci = [(^1] and C2 = {^2} for two Qrctl formulas tpi and (p2, then EU^ (Ci, C2) 
is l3'{^iUip2)j. 

We say that an equivalence relation ~ is l,p, EU -stable if, for all unions Ci, C2 of equivalence 
classes with respect to ~, and for all s,t & S with s ~ t, we have: 

(1) s G Pre(Ci) iff t G Pre(Ci); 

(2) s G Cpre(C7i) iff t G Cpre{Ci); 

(3) s G ^?7^ (Ci, C72) iff t G ^[/^ (Ci, C2). 

Let ~ATL ^'^^ coarsest equivalence relation that is 1, p, EU -stable. We show that ~atl 
coincides with r^-*^. 

Theorem 5.4. For all finite MDPs, we have ^^^L ~ ■ 
Proof. We prove containment in the two directions. 

~ATL — This statement is equivalent to saying that for all formulas ip in Qrctl'*'^, 

{ip} is the union of classes in S/ ~atl- ^ * states such that s i, and let ip 

be a formula from Qrctl^*' such that s \= ip and t ^ ip. We show by structural induction 
on ip that s t^atl ^- '^^^ cases where (p is a proposition, or the boolean combination of 
formulas are trivial. All other cases follow as in the proof of the first part of Theorem 14. 7^ 
except for the case ip = 3^ {p>iUp>2). For (p = 3^(931^^992), we have s G EU^{\ipi\, \p>2\): 
while t EU^ {Vpi\^ l¥'2l)- By inductive hypothesis, we can assume that \ip>i\ and [932I are 
unions of classes in S/ ~atl- (■^'0 ~atl- 

C ^f^i^. The proof follows the same idea of the proof of the first part of Theorem 14.41 
The only modification needed is in the inductive definition of the set of formulas: we take 
here *fc+i = BoolC(*fc U {3>o 0^,3^i)Ui)' \^,^' ^ *fc}). ■ 
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The following theorem provides an upper bound for the complexity of computing ~^ 
on MDPs. The PTIME-completeness of ordinary simulation |ABGS9l] provides a lower 
bound, but no tight lower bound is known. 

Theorem 5.5. The problem of deciding whether s t for two states s and t of an MDP 
is in co-NP. 

Proof. We show that the problem of deciding s 96^" t is in NP. To this end, we have to 
show that there is a certificate for s t that has polynomial size, and is polynomially 

checkable. Consider the usual partition-refinement method for computing ~atl |Mil90[ 
IAHKV98J . The method starts with an equivalence relation ~ that reflects propositional 
equivalence. Then, ~ is refined at most m = |S| times. At each refinement step, some 
state-pairs are removed from ~. A certificate for the removal of a pair from ~ is simply 
a Cpre or Pre or EU^ operator, along with a union of equivalence classes; it is thus of 
size polynomial in m. Since at most m? pairs can be removed from ~, the total size of 
these state-pair removal certificates is polynomial in m. This yields a polynomial-size and 
polynomially-checkable certificate for s t. I 



6. The Roles of Until and Wait-For 

In this section we study the roles of the until and the wait-for operator, and the relationship 
between the equivalences induced by Qrctl and Qrctl*. 

It is well known that in the standard branching logics Ctl and Ctl*, as well as in 
ATL, the next-time operator Q is the only temporal operator needed for characterizing 
bisimulation. For Qrctl, this is not the case: the operators U and W can increase the 
distinguishing power of the logics, as the following theorem indicates. 

Theorem 6.1. The following assertions hold: 

(1) On finitely-branching MDPs, we have r^-^'^'O = 

(2) For all MDPs, we have C f«>0'O. 

(3) For finite AMDPs, we have Ri>0'O = ~>o. 

(4) There is a finitely-branching, infinite AMDP on which Ri^'^'O ^ ss^*^. 

(5) There is a finite, (non- alternating) MDP on which k,^^'^ ^ w^*^. 

Proof. Assertion 1. The inclusion Ki^^^O c ~atl follows from the fact that formulas used 
in the first part of the proof of Theorem 14.41 make use only of the Q temporal operator, 
and from ~atl = ~atl- prove the inclusion ~atl ^ Ri^^'O, consider two states s,t ^ S 
such that s ^^^^'O Then, there is a Qrctl^'^ formula (/? that distinguishes them. Prom 
this formula we derive an Atl formula /((/?) that also distinguishes them. We proceed by 
structural induction. The result is obvious for boolean operators and atomic propositions. 
The cases if = 3^ Ov'i ^'^'^ V = Ovi easy consequence of Lemma 12.31 

Assertion 2. Immediate, as the set of Qrctl'*'^ formulas without U and W is a subset of 
the set of ah Qrctl'*° formulas. 

Assertion 3. The result is derived as follows: c ~atl = The inclusion 

~>0,O c RiATL is a consequence of Assertion [T] of this theorem. The equality ~Game = 
follows by combining Assertion [1] of Theorem 14.41 and Assertion [1] of Theorem 14.71 
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Assertion 4- The result follows by considering again the infinite AMDP of Figure [H Rea- 
soning as in the proof of Theorem 14.71 it holds (s,t) € ss^'^'O, but {s,t) ss^'^: indeed, 
note that s \= 3>^{nq) and t ^ 3>^{aq). 

Assertion 5. The result is a consequence of Theorem 14.71 Assertion [2l and of the present 
theorem, Assertion [1} the same MDP used to show ~atl 2 depicted in Figure O also 
shows ^>^'0 2 «>o. I 



7. Linear Time Nesting 

The logics Ctl and Ctl* induce the same equivalence, namely, bisimulation. Similarly, 
ATL and ATL* both induce alternating bisimulation. We show here that Qrctl and 
Qrctl* induce the same equivalences on finite, alternating MDPs, but we show that for 
infinite, or non-alternating, MDPs, Qrctl* induces finer relations than Qrctl. These 
results are summarized by the following theorem. 

Theorem 7.1. The following assertions hold: 

(1) For all MDPs, we have C f«>o. 

(2) For all finite AMDPs, we have Ri>° = f«>°. 

(3) There is a finitely-branching, infinite AMDP, on which f*^*^ ^ 

(4) There is a finite MDP on which ps>° ^ f»>°. 

Before presenting the proof of this result, it is useful to recall some facts about Rabin 
automata, Markov decision processes, and probabilistic verification. 

Rabin automata and temporal logic. An infinite-word automaton over AP is a tuple 
A = (L, Linit,'~-~', A), where L is a finite set of locations, Linu C L is the set of initial 
locations, : L 2^^ is a labeling function that associates with each location I G L the 
set '~P C AP of predicates that are true at I, and A : L 2^ is the transition relation. 
The automaton A is deterministic if the following conditions hold: 

• for all rj C AP, there is a unique / G Linu with '~P = rj; 

• for ah / G L and all r] C AP, there is G A(^) with = rj; 

• for ah /, /" G L, we have that /" G A{1) and /' / /" imphes ^l'^ / ^l"^. 

The set of paths of ^ is Paths{A) = {Zq, ^i, ^2, • • • Mo £ Linit^^k > O.lk+i G ^{h)}- A Rabin 
acceptance condition over a set L is a set of pairs F = {{Pi,Ri), (^2,-^2), • • • > {Pm,Rm)} 
where, for 1 < i < m, we have Pi,Ri C L. The acceptance condition F defines a set of paths 
over L. For a path r = sq, si, S2, ■ ■ ■ G L'^ , we define Inf(r) to be the set of locations that 
occur infinitely often along r. We define Paths{F) = {t £ \ 3i £ [l..m] . (Inf(r) D Pi = 
A Inf(r) n i?i 7^ 0)}. A Rabin automaton (A, F) is an infinite- word automaton A with set 
of locations L, together with a Rabin acceptance condition F on L; we associate with it the 
set of paths Paths {A, F) = Paths (A) n Paths (F). 

Given a set of predicates AP, a trace p G (2^^)'^ over AP is an infinite sequence 
of interpretations of AP; we indicate with Traces{AP) = (2'^^)'^ the set of all traces 
over AP. A Rabin automaton {A,F) with A = (L,Linit,'~-^,A) induces the set of traces 
Traces{A,F) = {'"/q"', '"^2'', • • • I lo,h,h,--- S Paths{A,F)}. An Ltl formula ip over 
the set of propositions AP induces the set of traces Traces{if) C Traces{AP), defined as 
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usual (see, e.g., |MP91] ). From [VW86j it is known that for an Ltl formula ip we can 
construct a deterministic Rabin automaton {A,F) such that Traces {A, F) = Traces {(p). 
We can now proceed to prove Theorem 17.11 

Proof of Theorem 17.11 

Proof. The first assertion is obvious. For the other assertions, we proceed as follows. 
Assertion 2. Let G = (5", ^,r,5, [•]) be a finite, alternating MDP. Since Qrctl is a frag- 
ment of Qrctl*, it follows that ^j"*^ C . To prove we show that if there 
exists a Qrctl* formula that distinguishes two states s and t, then there also exists a 
Qrctl formula that distinguishes s and t. We focus on formulas of the type 3^^Lp and 
3^(^, where is an Ltl formula. The generalization to the complete logic follows by 
structural induction and duality. Thus, assume that there are two states € S and 
a E {i,> 0} such that s* \= 3°(/9 and t* ^ Let {A^F) be a deterministic Rabin 
automaton such that Traces{A, F) = Traces{p>), and assume that A = (L, Linu, A) and 
F = {{Pi,Ri),...,{P^,R^)}. Let G' = GxA = (S' , A,T' ,[■]') be the MDP resulting 
from forming the usual synchronous product of G and A. In detail, we have: 

• S' = {(3,1) eSxL\[s]= ^P}; 

• r'{s,l) = r(s) for all (s,/) G S'; 

• for all (si, ^i), (s2, ^2) £ S' and a G ^, we have (5'((si, /i), a)(s2, ^2) = '5(si,a)(s2) if 
I2 G A(/i), and 6' {{si, h) , a){s2, h) = otherwise; 

• [(s, /)] = rp^ for all (s, /) G S'. 

Let F' be the Rabin acceptance condition of G', defined by F' = {{P[,R[), . . . , (-P^i -^m)}' 
where each P^, R[ C 5' is defined as follows: P[ = {{s, I) € S' \ I e Pi} and R'^ = {{s, I) e S' \ 
I G Ri}. For every s G S, denote with linit{s) the unique I G Linit such that [s] = ^V. Using 
the results of |dA97al ldAHK981 ICdAH04j on the model-checking of MDPs with respect to 
probabilistic temporal-logic properties, we can construct ^u-calculus formulas to distinguish 
(s*, ^jmt(s*)) and {t* , linit (t*))- Define, first of all, the following abbreviations: 

Pi n {Cpre{X) U {R'i n Cpre{Y))) 



i=l 
m 

= (J uY .iiX. 

i=l 



Pi n {Pre{X) U {R'i n Pre{Y))) . 

i=l 

On the basis of the above formulas, define: 

= . {'4)"^^ VJ Cpre{W)) 
=vZ .fiW . {Apre{Z,W)\J'4)^) 
iP>o = . UPre{W)) 

For a G {all, 1 ,> 0, some} and s G 5", we have: 



t/;-^" = \j uY. fiX. 



Pl n {Apre{Y,X) U (i?- n Cpre{Y))) 



^some ^ y nX. 
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SO that, in particular, {s* ,linit{s*)) G IV'"1g' (t* ,linit{t*)) ^ ['0"1g'- Hence, the formula 
is a ^-calculus witness, on G', of the distinction between s* and t* . We now show how to 
transform first into a /i-calculus formula to be evaluated on G, and then into a Qrctl 
formula to be evaluated on G. This will show that s* t*, as required. 

To obtain a /i-calculus formula on G, from ^p°' we construct a /i-calculus formula 7" 
with the following property: for all s G S, we have s € It^Ig (s,^imt(s)) G [^'"Ig'- "^^^ 
idea, taken from [dAHMOl], is as follows. 

First, ip'^ can be rewritten in equational form |BC96j . as a sequence of blocks B[, . . . , B'f^, 
where B[ is the innermost block and i?^ the outermost block. Each block Bp for 1 < j < k, 
has the form Vj = \ej, where A € {/i, j^}, and where ej is an expression not containing /x, 
I', in which all the occurrences of the variables vi, . . . ,Vk have positive polarity {BC96j : the 
output variable is Vk- 

From this formula, we obtain another formula 7", also in equational form, with sets of 
variables {vl \ I < i < k A I G L} U {vk+i}- Formula 7" simulates on G the evaluation of 

on G': for each variable Vi, with 1 < i < k, formula 7" contains the set of variables 
{u' I / € L}, where the value of Vi at location / G L is encoded as the value of at s. The 
formula ip consists of the blocks Bi, . . . , B^, plus an additional block -B^+i. For 1 < i < k, 
the block Bi contains the equations for the variables {vl | / E L}. The equation for vl is 
obtained from the equation for Vi as follows: 

• replace each variable Vi on the left-hand side with the variable vl; 

• replace Pj (resp. Rj), for 1 < j < m, with S if / € Pj (resp. / G Rj), and with if / ^ Pj 
(resp. I ^ Rj); 

• replace Cpre{vh), for variable 1 < h < k, with Cpre{[Ji,^^i^i-^ v\^); 

• intersect the right-hand side with Hgen"' 1 ^ ClqeAPXT 

The block Bk+i consists of only one equation Vk+i = U/eL,„,i '^fei ^^'^ ^e either a // or 
a z/-block. The output variable is Vk+i- 

The result of the above transformation is a /u-calculus formula 7" on G containing only 
the operators Cpre and Apre. By ()3.ip and Lemma 14.61 both operators can be encoded in 
Qrctl. Then, proceeding as in the first part of the proof of Theorem 14.71 we can "unroll" 
the computation of the fixpoints of the /i-calculus formulas, since we know that each fixpoint 
converges in at most IS*! iterations. The result of these two transformations is a Qrctl 
formula A", such that s* \= A" and t* ^ A", as required. 

Assertion 3. Consider the AMDP G with state space S = ({1,2,3} x N) U {0}. The only 
successor of is itself. States of the type (i,2n), for i G {1,2,3} (i.e., even states) belong 
to player 1, while odd states belong to player p. For all n > we have: r((l,2n)) = 
r((3,2n)) = {a,b} and r((2,2n)) = {a,b,c}, where, for ah i G {1,2,3}: 

Dest{{i,2n),a) = {{i,2n)} 

Dest{{i, 2n),b) = {{i, 2n + 1)} 

Dest{{2, 2n), c) = {(3, 2n + 1)}. 

Player p states starting with 1 or 2 lead to the next state in their chain and to the sink 
state with equal probability. Formally, r((i,2n + 1)) = {x} and 

,5((l,2n + l),x)((l,2n + 2)) = (^((l,2n + l),x)(0) = 

<5((2, 2n + 1), x)((2, 2n + 2)) = 6{{2, 2n + 1), x)(0) = 1/2. 
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Figure 9: An MDP where s s' and s s'. 



Finally, states starting with 3 obey the following distribution. 

6{{3, 2n + 1), x)((2, 2n + 2)) = exp(-l/2") 

(5((3,2n + l),x)(0) = 1 -exp(-l/2"). 

Observe that G is a finitely-branching, infinite AMDP. We take AP = {q}, and we ask 
that the predicate q be true at all odd states. Then, by induction on the structure of a 
Qrctl formula, it is not hard to see that (1,0) ss^*^ (2,0). On the other hand, we have 
(2,0) ^ 3>^n<}q and (1,0) ^ 3>^n<}q. 

Assertion 4- Consider the MDP depicted in Figure [9l By induction on the structure 
of a Qrctl formula, it is not hard to see that s s'. On the other hand, for ip = 
3^ {<}q A □3>° O q) we have s \= (f, s' ^ (p. M 

We do not provide an algorithm for computing on non-alternating MDPs. Identi- 
fying such an algorithm is an open problem. 
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